System and method for creating and manipulating synthetic environments

ABSTRACT

Disclosed herein are systems, computer-implemented methods, and tangible computer-readable media for synthesizing a virtual window. The method includes receiving an environment feed, selecting video elements of the environment feed, displaying the selected video elements on a virtual window in a window casing, selecting non-video elements of the environment feed, and outputting the selected non-video elements coordinated with the displayed video elements. Environment feeds can include synthetic and natural elements. The method can further toggle the virtual window between displaying the selected elements and being transparent. The method can track user motion and adapt the displayed selected elements on the virtual window based on the tracked user motion. The method can further detect a user in close proximity to the virtual window, receive an interaction from the detected user, and adapt the displayed selected elements on the virtual window based on the received interaction.

PRIORITY INFORMATION

This application is a continuation of U.S. patent application Ser. No.16/531,622, filed Aug. 5, 2019, now U.S. Pat. No. 11,064,136 which is acontinuation of U.S. patent application Ser. No. 13/600,652, filed Aug.31, 2012, now U.S. Pat. No. 10,375,320, which is a continuation of U.S.patent application Ser. No. 12/343,114, filed Dec. 23, 2008, now U.S.Pat. No. 8,259,178, all of which are herein incorporated by reference intheir entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to synthetic environments and morespecifically to virtual windows to a synthetic environment.

2. Introduction

People living in high density population centers often live in housesthat have few windows or have windows with views that are not veryaesthetically appealing. Offices in such areas are frequently part of ahigh-rise building where window offices are rare. People in suchconditions can go days on end without seeing sunlight except as theycommute to and from work. Human beings feel better when they can see theenvironment around them, including the sun, nature, the weather, clouds,and so forth. Certain people get depressed or feel miserable withoutsome contact with the outside. Moving away from a home or job to thecountry side may not be an option. This and other situations exist wherepeople want to see something other than their actual environmentalsurroundings or where people want others to see something other thanwhat is actually there. Accordingly, what is needed in the art is a wayto make windows that show something other than what is in fact on theother side of a wall.

SUMMARY

Additional features and advantages of the invention will be set forth inthe description which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. Thefeatures and advantages of the invention may be realized and obtained bymeans of the instruments and combinations particularly pointed out inthe appended claims. These and other features of the present inventionwill become more fully apparent from the following description andappended claims, or may be learned by the practice of the invention asset forth herein.

Disclosed are systems, computer-implemented methods, and tangiblecomputer-readable media for synthesizing a virtual window. The methodincludes receiving an environment feed, selecting video elements of theenvironment feed, displaying the selected video elements on a virtualwindow in a window casing, selecting non-video elements of theenvironment feed, and outputting the selected non-video elementscoordinated with the displayed video elements. Environment feeds caninclude synthetic and natural elements. The method can further togglethe virtual window between displaying the selected elements and beingtransparent. The method can track user motion and adapt the displayedselected elements on the virtual window based on the tracked usermotion. The method can alter one or more element of the environment feedover time. The method can further detect a user in close proximity tothe virtual window, receive an interaction from the detected user, andadapt the displayed selected elements on the virtual window based on thereceived interaction. The method can select elements based on a userprofile.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the invention can be obtained, a moreparticular description of the invention briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only exemplary embodiments of the invention and are not thereforeto be considered to be limiting of its scope, the invention will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1 illustrates an example system embodiment;

FIG. 2 illustrates an example method embodiment;

FIG. 3 illustrates an example configuration for virtual windows;

FIG. 4 illustrates a sample virtual window;

FIG. 5 illustrates an example environment feed aggregator;

FIG. 6 illustrates a building with coordinated external virtual windows;

FIG. 7 illustrates a system for receiving environment feeds andcoordinating their output;

FIG. 8 illustrates an example snap-in liquid crystal display (LCD)virtual window panel;

FIGS. 9A and 9B illustrate an example two-pane virtual window in an openand a closed position; and

FIG. 10 illustrates an example application of a nearby environment feed.

DETAILED DESCRIPTION

Various embodiments of the invention are discussed in detail below.While specific implementations are discussed, it should be understoodthat this is done for illustration purposes only. A person skilled inthe relevant art will recognize that other components and configurationsmay be used without parting from the spirit and scope of the invention.

With reference to FIG. 1, an exemplary system includes a general-purposecomputing device 100, including a processing unit (CPU) 120 and a systembus 110 that couples various system components including the systemmemory such as read only memory (ROM) 140 and random access memory (RAM)150 to the processing unit 120. Other system memory 130 may be availablefor use as well. It can be appreciated that the invention may operate ona computing device with more than one CPU 120 or on a group or clusterof computing devices networked together to provide greater processingcapability. A processing unit 120 can include a general purpose CPUcontrolled by software as well as a special-purpose processor. An IntelXeon LV L7345 processor is an example of a general purpose CPU which iscontrolled by software. Particular functionality may also be built intothe design of a separate computer chip. A STMicroelectronics STA013processor is an example of a special-purpose processor which decodes MP3audio files. Of course, a processing unit includes any general purposeCPU and a module configured to control the CPU as well as aspecial-purpose processor where software is effectively incorporatedinto the actual processor design. A processing unit may essentially be acompletely self-contained computing system, containing multiple cores orCPUs, a bus, memory controller, cache, etc. A multi-core processing unitmay be symmetric or asymmetric.

The system bus 110 may be any of several types of bus structuresincluding a memory bus or memory controller, a peripheral bus, and alocal bus using any of a variety of bus architectures. A basicinput/output (BIOS) stored in ROM 140 or the like, may provide the basicroutine that helps to transfer information between elements within thecomputing device 100, such as during start-up. The computing device 100further includes storage devices such as a hard disk drive 160, amagnetic disk drive, an optical disk drive, tape drive or the like. Thestorage device 160 is connected to the system bus 110 by a driveinterface. The drives and the associated computer readable media providenonvolatile storage of computer readable instructions, data structures,program modules and other data for the computing device 100. In oneaspect, a hardware module that performs a particular function includesthe software component stored in a tangible computer-readable medium inconnection with the necessary hardware components, such as the CPU, bus,display, and so forth, to carry out the function. The basic componentsare known to those of skill in the art and appropriate variations arecontemplated depending on the type of device, such as whether the deviceis a small, handheld computing device, a desktop computer, or a computerserver.

Although the exemplary environment described herein employs the harddisk, it should be appreciated by those skilled in the art that othertypes of computer readable media which can store data that areaccessible by a computer, such as magnetic cassettes, flash memorycards, digital versatile disks, cartridges, random access memories(RAMs), read only memory (ROM), a cable or wireless signal containing abit stream and the like, may also be used in the exemplary operatingenvironment.

To enable user interaction with the computing device 100, an inputdevice 190 represents any number of input mechanisms, such as amicrophone for speech, a touch-sensitive screen for gesture or graphicalinput, keyboard, mouse, motion input, speech and so forth. The input maybe used by the presenter to indicate the beginning of a speech searchquery. The device output 170 can also be one or more of a number ofoutput mechanisms known to those of skill in the art. In some instances,multimodal systems enable a user to provide multiple types of input tocommunicate with the computing device 100. The communications interface180 generally governs and manages the user input and system output.There is no restriction on the invention operating on any particularhardware arrangement and therefore the basic features here may easily besubstituted for improved hardware or firmware arrangements as they aredeveloped.

For clarity of explanation, the illustrative system embodiment ispresented as comprising individual functional blocks (includingfunctional blocks labeled as a “processor”). The functions these blocksrepresent may be provided through the use of either shared or dedicatedhardware, including, but not limited to, hardware capable of executingsoftware and hardware, such as a processor, that is purpose-built tooperate as an equivalent to software executing on a general purposeprocessor. For example, the functions of one or more processorspresented in FIG. 1 may be provided by a single shared processor ormultiple processors. (Use of the term “processor” should not beconstrued to refer exclusively to hardware capable of executingsoftware.) Illustrative embodiments may comprise microprocessor and/ordigital signal processor (DSP) hardware, read-only memory (ROM) forstoring software performing the operations discussed below, and randomaccess memory (RAM) for storing results. Very large scale integration(VLSI) hardware embodiments, as well as custom VLSI circuitry incombination with a general purpose DSP circuit, may also be provided.

The logical operations of the various embodiments are implemented as:(1) a sequence of computer implemented steps, operations, or proceduresrunning on a programmable circuit within a general use computer, (2) asequence of computer implemented steps, operations, or proceduresrunning on a specific-use programmable circuit; and/or (3)interconnected machine modules or program engines within theprogrammable circuits.

Having disclosed some elementary system components, the disclosure turnsto the example method for creating and manipulating syntheticenvironments, as illustrated in FIG. 2. For simplicity, the method isdiscussed in terms of a system configured to practice the method. Thesystem first receives an environment feed (202). An environment feed caninclude synthetic and natural elements. An environment feed is somewhatlike a video feed of a conference or a meeting, but also includesextra-audiovisual elements. For example, an environment feed of afootball game can include multiple video angles, sound recordings frommultiple positions, smell, humidity, wind speed and direction,background noise, amount and color of ambient light, and so forth. Inone aspect, environment feeds also include still images, such as familypictures, political ads, and movie posters. In one aspect, users pay forenvironment feeds at varying prices. For instance, an environment feedof the pink beaches in Bermuda can cost more than a feed of the NewJersey shore. Different feeds can cost different prices at differenttimes. For example, environment feeds of New Orleans during Mardi Grasor Times Square at New Years Eve can cost more than at other times. Inone aspect, an advertiser can pay the owner of a system to displayadvertiser-selected environment feeds, perhaps in high traffic areas.

The system selects video elements of the environment feed (204). Thesystem can select elements that are all or part of an environment feed.The system can select a specific object or exclude a specific objectfrom the feed. For example, in an environment feed of a golf course thesystem can select the visible golf course and exclude the highway in thedistance behind the golf course. The system then displays the selectedvideo elements on a virtual window in a window casing (206). A virtualwindow in a window casing can be an LCD or plasma screen in a panelmounted inside a wall. A virtual window appears to be a standardtransparent window, but can display things other than what a standardwindow would show. The virtual window can be on the interior or exteriorof a structure, vehicle, or building. An interior virtual windowdisplays an image of what is or what appears to be the outside. Anexterior virtual window displays an image of what is or what appears tobe the inside. An interior and exterior virtual window can beincorporated into one unit. In the case of LCD panels that require abacklight, an interior and exterior virtual window can share a backlightelement. When an environment feed includes multiple visual angles orperspectives, the system can coordinate multiple virtual windows in asingle room for a visually consistent appearance. For instance, in aroom with virtual windows to the north, south, and east, the system candisplay environment feeds originating from the same point, but recordingdifferent directions so a person who views the virtual windows sees aplausible place.

The system selects non-video elements of the environment feed (208) andoutputs the selected non-video elements coordinated with the displayedvideo elements (210). In one aspect, the system retrieves a user profileand selects video and/or non-video elements of the environment feedbased on the user profile. For example, if a user has an extremelysensitive sense of smell, that user's profile can indicate never toinclude smells. The system can coordinate output of these non-videoelements with playback devices incorporated into the virtual window orin the vicinity of the virtual window. For example, the frame of thevirtual window can include speakers, scent emitters, fans, humidifiers,dehumidifiers, and so forth. The system can coordinate with devices notincorporated in the virtual window, such as ceiling fans or standardroom lighting. One example non-video element is ambient lighting. Thesystem can coordinate the room lighting to darken, brighten, flash, orchange colors to match the displayed environment feed.

In one aspect, the system can toggle virtual windows between displayingthe selected elements and being transparent. Transparency can cover theentire surface of the virtual window or portions of the virtual window,allowing users to see a mix of virtual window elements and what isactually on the other side of the virtual window. The transparency canbe complete, i.e. 100% transparent, or partial, i.e., 25% transparent.

Further, the system can alter one or more element of the environmentfeed over time. For example, if an environment feed depicts a tree, inspring the system displays the tree with leaf buds. As the summerprogresses, the system displays the tree with branches full of greenleaves. As autumn approaches, the system displays leaves changing colorand falling to the ground. The system can alter elements on a seasonal,annual, monthly, daily, or hourly basis or at any other time increment.The system can also alter non-video elements over time.

In another aspect, the system tracks user motion and adapts thedisplayed selected elements on the virtual window based on the trackeduser motion. The system can do this to provide the rendering depth andappearance of a real window as the user moves about the room. As theperspective and/or position of the user changes, what the user would seein a real window changes. The system updates the virtual window toimitate those changes. The system can respond to any user motion, largeor small. One example of this is head tracking. The system can locateand track the position and/or orientation of a user's head in the roomand adapt the display so the perspective displayed on the virtual windowrenders depth as a real window would. This effect can require a basicthree dimensional modeling of the contents of the environment feed. Thesystem can use cameras and/or distance sensors (otherwise known as rangefinders) to calculate the movement and position of a person (or herhead). Some commonly used distance sensors are infrared sensors that usereflected light, triangulation based sensors, and ultrasonic sensors.The system can depend on some kind of reflector or transmitter carriedor worn by users to detect their position and/or identity.

FIG. 3 illustrates an example configuration for virtual windows. Thisconfiguration 300 is exemplary. Other configurations exist. The room 300has four walls. One side has an interior virtual window 304 and acorresponding exterior virtual window 306. Both the interior and theexterior virtual windows have corresponding microphones 308, 312 andcameras 310, 314. In addition to more traditional sensors such asmicrophones and cameras, the system can also include sensors and outputtargeted at different senses. For example, a wind sensor and a windgenerator, a smell sensor and smell generator, or a temperature sensorand temperature generator. A user can individually disable or enableeach of the various inputs completely or in degrees. In a situationwhere users purchase environment feeds, different subsets of thecomplete sensed environment can be priced differently. For example, afree environment feed can include only the visual portion of the feed.For an additional fee the environment feed can include sound, foranother additional fee the environment feed can include smell, and soon. A viewer 316 on the inside of the room can see one image on theinterior virtual window 304 while a viewer 318 on the outside of theroom can see a different image on the exterior virtual window 306. Aserver can augment the video signal of the interior camera 308 withsynthetic elements to display on the exterior virtual window 306,allowing the viewer 318 to see a modified version of what is behind theexterior virtual window 306, as if it were a real window. For instance,the server can modify the video signal to display people in the houseeven when no one is home. The server can modify the video signal todisplay a fireplace against the wall when there is none. Similarly, theinterior virtual window 304 can display an environment feed of afar-away location or of the exterior camera 314. The server can augmentvideo, synthesize elements for inserting into a video, or display acompletely synthetic environment. For example, if the virtual windowwere a real window, the inside viewer 316 may see a dark alleyway,whereas the inside viewer 316 can see an environment feed of a forest, aglacier, a park, golf course, or other desirable location on the virtualwindow. A user can set up an environment feed of her house for displayon a virtual window at work. The system can include other outputs thanthe visual aspect of a virtual window, such as vibration, sound, smell,tactile output, windiness, humidity, and temperature. The server forcontrolling the environment feed can be integrated as part of thevirtual window, can be a locally connected module (wired or wireless)which coordinates the output at each virtual window, or the server canbe a centralized network server which operates in the Internet or acomputing cloud.

One side has a single display that operates as both an interior and anexterior virtual window 320. In this case, both the inside viewer 316and the outside viewer 318 see the same image on the virtual window. Thevirtual window can simulate wooden blinds or a bug screen, for example,where parts of the virtual window are opaque and parts are transparent.The system can tint virtual windows entirely or partially to blocksunlight or to prevent others from peeping inside the room.

One side has only an interior virtual window 322 which is hung on a walllike a painting. The system can coordinate this interior virtual window322 with the other displays 320, 304. While the system could coordinatethis interior virtual window 322 with the exterior virtual window 306,it may not make sense in this context because each targets differentviewers. However, if the virtual window 320 is partially transparent andshows snow falling outside, the system can coordinate the other interiorvirtual windows 304, 322 by altering their feeds to include snow orother indications of cold weather. Other exterior weather and conditionscan influence how the system modifies related environment feeds.

The system can index each addressable output device and theircapabilities. The system can form groups of output devices to update andmaintain in synchronization for output. Output devices can belong to oneor more group. In this case, the system can assign groups differentpriority levels. In one aspect, the system only displays the environmentfeed associated with the highest priority group. In another aspect, thesystem blends environment feeds of different priorities where possibleand possibly in proportion to their priority.

FIG. 4 illustrates a sample virtual window. The house 400 includes awall 402 with a virtual window 404 in place of a normal window. Thevirtual window 404 can either be integrated into the wall 402 like awindow frame or separate from the wall like a picture hanging on a wall.The virtual window 404 is on the interior side of the wall 402 andtypically displays simulated views of what can be on the other side ofthe wall or augmented views of what actually is on the other side of thewall. A simulated view can be a grassy field with fluffy white clouds inthe sky. An augmented view can be adding a rainbow to an existingoutside view or showing leaves on the trees even after they have reallyfallen down, for example. A user can install any of the typical windowdressings 406 such as blinds, curtains, or drapes in front of a virtualwindow so the virtual window does not appear unusual. A user can installor place auxiliary output devices, such as speakers 410, somewhere inclose proximity to the virtual window. A user can install other inputdevices, such as a camera or microphone 408 to record one side of thevirtual window for output on the opposite side. The system can use thevideo camera 408 to transform the virtual window into a mirror-likedisplay so the user can see a reflection of himself or herself and whatis in the room. The system can include wireless headsets connected tothe speakers to allow the user to experience and enjoy the soundswithout disturbing others. The system can also incorporate other sensesin similar ways.

FIG. 5 illustrates an example environment feed aggregator. Varioussources 502, 504, 506 provide environmental feeds of differentlocations, such as a tropical island, a playground, or a golf course.Environmental feeds can include video, audio, smell, weatherinformation, temperature, humidity, and so forth. An environment feedaggregator 508 gathers and stores these environmental feeds 502, 504,506. The aggregator can be part of a system controlling virtual windows510 or it can be separate. A system controlling windows can request orsubscribe to environment feeds for individual virtual windows or sets ofvirtual windows. Environment feeds can include multiple channels of thesame type of information, such as three different camera angles of thesame location. The environment feed aggregator 508 can send and receiveenvironment feeds via closed-circuit cameras, video over IP, storedvideo on a hard drive, and other formats. The aggregator 508 can provideenvironment feed subscription services, like Really Simply Syndication(RSS). The aggregator can pre-combine elements of different environmentfeeds to generate new feeds. Environment feeds can include partialinformation, such as only a smell, or only audio, or only a portion of avideo which can be laid over an existing feed such as drapes or windowshades.

People operating environment feeds in desirable or beautiful areas cansell or license their feeds to others. For example, a person in Tahitican license their feed of the beach. An Indian reservation can selltheir feed of the rim of the Grand Canyon. The International SpaceStation can offer for free their feed of the Earth from space tocontributors or employees of the various space agencies while charging afee for others. In the case of a feed that is from a different timezone, the system can time-delay the feed so the time of day shown in thefeed matches the local time of day.

FIG. 6 illustrates a building with coordinated external virtual windows.A central server can coordinate the external display of multiple virtualwindows. Multiple virtual windows can operate as a single, macro leveldisplay. For instance, a high-rise 602 with virtual windows can displaya large image of a single environment feed. The system can configure theexternal virtual windows to display coordinated advertising material,essentially transforming the building into a large billboardadvertisement. The external virtual windows can also display a texturesimilar to the surrounding building material in order to blend in andnot appear to be a window.

FIG. 7 illustrates a system for receiving environment feeds andcoordinating their output. The system 700 receives environment feeds 702at a server 704. Environment feeds can come from remote places or can belocal, such as a nearby camera or a camera integrated into the oppositeside of a virtual window. The server 704 also retrieves a user profile706 including user preferences and personalization information. Theserver selects one or more environment feeds 702 and assembles naturaland synthetic portions together for output on one or more virtualwindows 708. Environment feeds can also include non-visual outputs, suchas audio 710, a fan 712 to create a breeze, and a humidity or smell 714generator. The server 704 can control and coordinate each of theseoutputs to simulate a particular environment feed. One exampleenvironment feed is a tropical island. The server 704 can simulate atropical island by displaying a blue, sunny sky, palm trees, and thebeach in multiple virtual windows 708, playing the sound of a breeze andthe ocean over speakers 710, turning on fans to provide an actual breezein the room 712, and providing a slight humidity and the smell of saltysea air 714. The system can also add synthetic elements to the feed. Forexample, if the user has a database of family photographs or home videosat the beach, the system can insert or merge all or part of thoseelements into the virtual windows. The system can also incorporatepurely synthetic elements as well, such as a three-dimensional model ofa seagull flying in the sky. If the user wants to simulate the effect ofbeing on a boat, the system can gently rock images in each window backand forth.

FIG. 8 illustrates an example snap-in LCD virtual window panel. A wall800 has a recessed section holding a virtual window panel in a mannersimilar to an actual window. Similar to a double-paned glass window, thevirtual window panel has two faces 802, 804. One face 802 displays aninterior image and one face 804 displays an exterior image. Additionallayers can exist over the virtual window panel. For example, a glass orclear plastic panel can protect the virtual window panel. Drapes, aninsect screen, or a lattice of wood (or faux wood) can partially obscurethe virtual window panel to make it look more like an authentic window.A single, user-replaceable unit having two LCD virtual window panels anda shared backlight element can snap or latch in place. Panels can alsobe held in place magnetically. When a panel snaps in place, the windowpanel automatically connects with contacts for electricity 806 as wellas a video source 808, such as an environment feed or a local feedserver. The automatic connection can be similar to the electricalcontacts on a video game cartridge, secure digital card, or a RAM modulein a computer. In one aspect, rather than replacing an entire unit,users can replace just the LCD panels or the backlight element withinthe virtual window panel. In one aspect, a solar panel can provide allor some of the power to LCD panel and/or the backlight element. WhileLCD displays are discussed, other suitable display technology can beused as well, such as organic light emitting diode (OLED), plasma,digital light projection (DLP), and cathode ray tube (CRT) displays.

FIGS. 9A and 9B illustrate an example two-pane virtual window in an openand a closed position. Two or more virtual windows can operate inconcert to show different portions of the same environment feed. Onesimple example of this is a two-pane virtual window 900 made up of twoLCD panels, one on top 902 and one on bottom 904. Much like a realwindow, the bottom panel slides open to view what is behind the virtualwindow 906. As the bottom panel slides up, sensors detect the positionof the bottom pane 904 in relation to the top pane 902 and the systemadjusts the image on the bottom pane 904 to match what is on the toppane 902. The result is a smooth, uninterrupted image spanning bothpanes 902, 904 even though the bottom panel 904 is moved. The sensorsfor this can be in the panes themselves or in the wall or in a windowsill. In one aspect, the user can indicate that when the window pane isopen, the system disables the environment feed or makes it match what isbehind the virtual window 906. Similarly, as one pane is closed, thesystem detects the position of the panes and smoothly updates thevirtual windows to display a unified image across the two.

FIG. 10 illustrates an example office building application of a nearbyenvironment feed. In this example, an office building 1000 has tworooms. One room on the inside 1002 does not have an external wall andone room on the outside 1004 has an external wall with an outward facingwindow. The worker 1006 in the inside room 1002 cannot see outside.However, the worker 1008 in the outside room 1004 can see outside. Avirtual window 1010 mounted on or in the wall can display a feed from acamera 1012 only a few meters away. The camera 1012 can be mounted justabove or below the window of the outside room 1004 to display to theworker 1006 in the inside room 1002 substantially the same view as theworker 1008 in the outside room 1004. In this way, each office in abuilding can share the corner office benefit of a great view. The worker1006 can select different external feeds for the virtual window 1010,such as feeds of the corporate headquarters, of a different officewindow, or of a golf course.

When workers stay late, the system can delay playback to “extend” theday to match the later hours. For example, an intermediate recordingmodule, not shown, can record the view through the camera 1012throughout the day. The system can calculate at what rate to extendplayback to fill the late hours and play the recording back at a slowerspeed. In this way, if a worker stays until 7:30, the virtual windowshows the environment feed more and more out of sync with the actualoutside environment, but in sync with the worker's routine of seeing thesun go down just as he leaves the office. This technique can beparticularly effective if the worker does not see any conflicting visualcues, such as a real window to the outside of the building.

In another embodiment, a virtual window can replace a wall panel or aceiling panel. Instead of displaying an environmental feed of somewhereelse or somewhere exotic, a virtual window can show a pattern simulatingwallpaper, a ceiling tile, or even a fluorescent light panel. To aid inmaintenance or diagnostics, a user can toggle the virtual window to showwhat is immediately behind it. For example, the virtual window cancontain a camera which allows the virtual window to display the variousduct work, plumbing, and wiring located behind a wall or in the plenumspace above a ceiling. The same principle can be applied to things likecomputer cases or body panels on a car to allow for the virtual windowor virtual panel to display one texture, image, or pattern and toggle toa view of what the panel covers. Such functionality allows for a quickglance at what is behind the panel, as if the panel were transparent ornot there at all.

In another aspect where virtual windows include a camera to see what isbehind the window, the system can allow users to remotely log in to thewindow to view the camera feed. For example, a user can log on remotelywith a computer, remote control, or smart phone to a virtual window inthe front of her house to remotely see what is happening there. The usercan log in and view that feed from within the house or halfway aroundthe world.

In one aspect, virtual environment feeds includes three-dimensionaland/or formable displays which are either flexible or rigidly molded toa non-flat shape (i.e. circular, spherical, wavy, curved, ridged, bumpy,or otherwise contoured to the shape of a real or imaginary physicalobject). For instance, legs of a char can be made of or fitted with acurved, tubular virtual window. The chair leg virtual window can displaya mahogany wood textured image for formal occasions and a simple whiteor beige textured image for other occasions. In another example, a largevirtual window is placed underneath a clear plastic or glass kitchencountertop. As the kitchen owner's mood changes, she can change thevirtual window texture to display things like running water, granite ormarble countertop texture, or an environment feed of the childrenplaying in the backyard. A user can display environment feeds of variousplaces around the house and surrounding grounds to monitor children, forhome security purposes, to know when guests are arriving, etc. Thesystem can act as a surveillance system in this regard.

The system can overlay synthetic elements on top of any of these feeds,such as a television show, clock, weather forecast indicators, or otherelements. In one aspect, users can interact with the countertop usingspeech, gestures, touch, plates, utensils, food, clapping, or otherinput. One such interaction is to toggle between environment feeds, suchas between the front of the house to see when a pizza delivery personarrives and the backyard to see what the children are up to. In anotheraspect, one large or multiple small virtual windows can attach to aceiling to simulate the sky, such as birds flying overhead, the movementof the sun, moon, stars, comets, clouds, airplanes, planets, and otherairborne (or space borne) objects).

In one aspect, to conserve power and possibly the life of the virtualwindow elements, the system can communicate with or incorporate motionsensors. When the system detects a certain period of time with nomotion, the system can turn off one or more virtual window or place itin a reduced power mode. The system can use different thresholds ofmotion sensitivity and time without motion depending on time of day, dayof the week, or can be manually set by a user.

In one aspect where the system is connected to security sensors, thesystem can coordinate displays with detected break-ins or breaches. Forexample, if a burglar breaks in a house, the system can display imagesand play sounds on the interior and/or exterior facing virtual windowsintended to frighten the burglar into retreating. One example isdisplaying on exterior windows an image of the homeowner with a shotgunor other menacing weapon. Another example is displaying on an exteriorwindow a video of a large guard dog such as a Mastiff, German Shepherd,Pit Bull, or Rottweiler accompanied with loud, ferocious barking. Yetanother example is displaying on interior virtual windows an image of apolice car pulling up in the driveway with accompanying sirens andflashing lights. The system can adapt the images on the virtual windowsto match the time of day, the weather, shadows, reflections, and otherfactors. When the security alarm is tripped, the system can takepictures of the intruder(s) and display them on the virtual windows witha text or spoken message that these pictures have been sent to thepolice and the police are coming quickly to investigate (whether that istrue or not).

In another useful aspect, a system of multiple external virtual windowscan connect with a fire alarm system. The fire alarm system detects afire in a specific location or room and communicates that location tothe system of virtual windows. To assist firefighters in finding thecorrect house, finding the fire within the house, or in locating personsinside the house, the system can display images on the external windows.For example, if the fire is in one part of the house, the system candisplay large flashing red images of fire on the nearest externalwindows. If a person is trapped in an upstairs room and cannot exit theburning house, the system can display images of a person or a messageindicating a person is still in the house. Firefighters can then plantheir rescue and firefighting efforts based on exterior window messages.Electricity may or may not be available from the main household powerlines during a fire. Virtual windows can include a limitedself-contained or local power source independent of the main householdpower, such as a battery backup, a solar panel, or a hand crank.

Virtual windows can display seasonal decorations. For example, exteriorwindows can display blinking red and green lights at Christmas time,spider webs and pumpkins at Halloween, or flag banners at the Fourth ofJuly. Interior windows can augment the actual outside view with snow atwinter time, a star in the eastern sky at Christmas time, wanderingtrick-or-treaters in costume on Halloween, or fireworks on the Fourth ofJuly. The system can accompany any of these variations with appropriatesounds and/or other sensory outputs, such as vibration, smell, wind, orhumidity. In a slightly more detailed variation, exterior windows canshow a view of the inside of the house at Christmas time whichdynamically changes. For example, a Yule log burning in the fireplacecan burn down throughout the course of the evening. Or, over a longerperiod of time, the window can display a Christmas tree. As days passand Christmas gets closer, the window can display the Christmas treewith more decorations, with more presents underneath it, etc. The windowcan tweak the tree's appearance to droop as it stands in the house andcan even show needles piling up under the tree. On Christmas morning,the virtual display can show people opening presents and playing withtoys. After Christmas, the virtual display can show the Christmas treeturning brown, decorations falling down, wrapping paper all over thefloor, and other post-Christmas images. These actions and their timingcan be user-customizable.

During election times, a user can set the virtual windows to displaypolitical campaign materials in a manner similar to bumper stickers orlawn signs. In one aspect, the system can include a database of localsound and advertising ordinances. The system can warn users if theirdesired actions would violate sound and/or advertising ordinances.

In one embodiment, virtual windows can toggle between being transparent(thereby functioning as an actual window would) and displaying an imageor video. Virtual windows can be transparent in sections, such as asimulation of frost in the corners of the window. The virtual windowscan provide partial transparency to provide some amount of shade,similar to a tinted window. The system can maintain a certain level ofsunlight in the room using dynamically tinting, partially transparentvirtual window.

Users can opt to display advertisements on virtual windows facing theexterior. Advertisements can be overlaid on top of a virtual windowdisplay, fill an entire virtual window, or can span multiple coordinatedvirtual windows. Homeowners can select the types of advertisements. Forexample, a single male may not want his house to show advertisements forlipstick or Hannah Montana. He can opt for advertisements for productshe favors such as the NFL or a Hummer H1. Another homeowner can selectthe highest-paying advertisements regardless of content. Homeowners canindicate the content, amount, duration, timing, and placement ofadvertisements on their external and/or internal virtual windows.

In one embodiment, virtual windows include or are connected to sensorswhich detect the proximity of known individuals. The system can displaydifferent views based on who is nearby. One way to track individuals isby cell phone signals or radio frequency identification (RFID). Thesystem can also transform exterior virtual windows on structures likehomes or businesses into billboards to display targeted advertising topeople driving by. For example, the system can detect a personapproaching the house. The system retrieves and displays advertisementson exterior virtual windows based on the preferences and interests ofthe person driving by. After the person drives past the house, thesystem resumes displaying the previous environment feed.

Homeowners can choose to display on exterior windows images of theinside of their house with everything neat, clean, and in order, when infact the house is dirty and messy on the inside. Alternatively,homeowners can show their family sitting down to watch television whenthey are away to give the appearance that someone is home. In order toconvincingly portray this appearance, the system can modify aprerecorded video feed to adjust clocks to the proper time, to displaycurrent television show programming on the television in the image, andso forth. The virtual windows can display natural elements, syntheticelements, or a combination of natural and synthetic elements. Exteriorvirtual windows can display to outsiders images of a clean and neathouse with expensive, luxurious furniture and architecture which is notactually on the inside of the house. The system can personalize virtualwindows based on proximity of persons inside or outside the building.For example, the system can identify residents currently inside thehouse and dynamically change the views based on their profile or tastes.The system can identify people using face recognition/detection, speakeridentification, RFID or other wireless tracking signals from portableelectronic devices, etc. The system can identify unrecognizedindividuals and display environment feeds selected by the homeowner forunfamiliar people.

A set of virtual windows can coordinate displays and timing to assistgraveyard shift workers to adjust to working at night by tricking theirbodies into thinking it is daytime at night and nighttime during theday. Similarly, virtual windows can gradually shift their timing overseveral days to help travelers ease the adverse effects of jet lag.

In one aspect, the system controlling virtual windows connects with ahome theater. As the home theater displays a movie or television show,the system can adjust the interior virtual windows to reflect theweather, surroundings, and ambiance of what the home theater isdisplaying. For example, if the home theater is showing the cantinascene in the original 1977 Star Wars® movie, the system can synthesize adark, smoky, alien-filled bar on Tatooine to display on the virtualwindow. The system can generate an entirely synthetic display for thevirtual window, or the system can use clips, pieces, and other mediafrom Star Wars to generate the appropriate output on the virtual window.In another example, the user is playing an Incredible Hulk video game onthe home theater. The system can extract portions of the Incredible Hulkmovie to display on the virtual window to round out the media atmospherein the room. This system can extend this concept to the external windowsas well. For instance, external windows can display to people outsidewhat movie you are watching or what game you are playing. This aspect ofthe invention provides some aspects of a drive-in theater, but onwindows or walls of a house rather than one large projection screen.

The system can include a module to transcodes environment feeds to fitthe desired output. Virtual windows can be of virtually any resolutionand indeed are likely to occur in varying resolutions according to sizeof the virtual window, cost, viewing distance, and other factors. Thesystem can convert a very high resolution environment feed down to alower bit rate or to a lower resolution depending on network needs oroutput device capabilities. Similarly, the system can apply a smoothingalgorithm to scale small video or photos to a higher resolution withoutblocky resolution artifacts. The system can adapt 5.1 surround sound tostereo sound if the output device does not support full channel surroundsound. In case the environment feed is stored or encoded in a formatwhich the virtual window cannot recognize, the system can stream orconvert the environment feed to a recognizable format for the virtualwindow.

In one embodiment, external virtual windows are subject to some level ofcommunity control. For instance, neighbors can provide feedback or inputto shut down or change the environment feed of a virtual window thatoutputs offensive, obscene, or questionable content. Neighbors can sendfeedback to the system controlling the exterior virtual window viaemail, Short Message Service (SMS), remote control, and other ways.

In another embodiment, virtual windows and environment feeds can be partof virtual vacations. Virtual vacation systems using virtual windows canbe implemented as part of a house or as a service, such as a spa. Forexample, a person can visit a virtual vacation salon where the user canpay a fee per hour to sit on a balcony virtually overlooking theMediterranean and reading a book. A person in Arkansas can use virtualwindows and environment feeds to make the inside of their house look andfeel like a ski lodge in Colorado. The costs associated with these feedscan be based on desirability, demand, expense in recreating theenvironment feed, number of sensory elements required for the feed, andso forth.

Embodiments within the scope of the present invention may also includecomputer-readable media for carrying or having computer-executableinstructions or data structures stored thereon. Such computer-readablemedia can be any available media that can be accessed by a generalpurpose or special purpose computer, including the functional design ofany special purpose processor as discussed above. By way of example, andnot limitation, such computer-readable media can comprise RAM, ROM,EEPROM, CD-ROM or other optical disk storage, magnetic disk storage orother magnetic storage devices, or any other medium which can be used tocarry or store desired program code means in the form ofcomputer-executable instructions, data structures, or processor chipdesign. When information is transferred or provided over a network oranother communications connection (either hardwired, wireless, orcombination thereof) to a computer, the computer properly views theconnection as a computer-readable medium. Thus, any such connection isproperly termed a computer-readable medium. Combinations of the aboveshould also be included within the scope of the computer-readable media.

Computer-executable instructions include, for example, instructions anddata which cause a general purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions. Computer-executable instructions also includeprogram modules that are executed by computers in stand-alone or networkenvironments. Generally, program modules include routines, programs,objects, data structures, components, and the functions inherent in thedesign of special-purpose processors, etc. that perform particular tasksor implement particular abstract data types. Computer-executableinstructions, associated data structures, and program modules representexamples of the program code means for executing steps of the methodsdisclosed herein. The particular sequence of such executableinstructions or associated data structures represents examples ofcorresponding acts for implementing the functions described in suchsteps.

Those of skill in the art will appreciate that other embodiments of theinvention may be practiced in network computing environments with manytypes of computer system configurations, including personal computers,hand-held devices, multi-processor systems, microprocessor-based orprogrammable consumer electronics, network PCs, minicomputers, mainframecomputers, and the like. Embodiments may also be practiced indistributed computing environments where tasks are performed by localand remote processing devices that are linked (either by hardwiredlinks, wireless links, or by a combination thereof) through acommunications network. In a distributed computing environment, programmodules may be located in both local and remote memory storage devices.

The various embodiments described above are provided by way ofillustration only and should not be construed to limit the invention.For example, the principles herein may be applied to internal andexternal windows of structures and vehicles as well as other surfaceswhere dynamic virtual displays are desired. Those skilled in the artwill readily recognize various modifications and changes that may bemade to the present invention without following the example embodimentsand applications illustrated and described herein, and without departingfrom the true spirit and scope of the present invention.

We claim:
 1. A method comprising: identifying, by a processor, anenvironment feed from a plurality of environment feeds, wherein eachenvironment feed of the plurality of environment feeds representsphysical environmental characteristics associated with a differentphysical location; and toggling, by the processor, a virtual window in awindow casing between displaying at least one video element of theenvironment feed and having a transparency ranging from zero percenttransparent to one hundred percent transparent.
 2. The method of claim1, further comprising: outputting at least one non-video elementcoordinated with the at least one video element.
 3. The method of claim1, wherein the environment feed comprises synthetic and naturalelements.
 4. The method of claim 1, further comprising: tracking a usermotion; and adapting the at least one video element on the virtualwindow based on the user motion.
 5. The method of claim 1, furthercomprising: altering at least one element of the environment feed over aperiod of time.
 6. The method of claim 1, further comprising: detectinga user in close proximity to the virtual window; receiving aninteraction from the user; and adapting the at least one video elementon the virtual window based on the interaction.
 7. The method of claim2, further comprising: retrieving a user profile; and selecting at leastone of: the at least one video element and the at least one non-videoelement of the environment feed based on the user profile.
 8. The methodof claim 1, wherein the environment feed comprises at least one of: asound element, a video element, a smell element, a humidity element, atemperature element, and a wind element.
 9. A system comprising: aprocessor; and a non-transitory computer-readable storage medium havingstored therein instructions which, when executed by the processor, causethe processor to perform operations, the operations comprising:identifying an environment feed from a plurality of environment feeds,wherein each environment feed of the plurality of environment feedsrepresents physical environmental characteristics associated with adifferent physical location; and toggling a virtual window in a windowcasing between displaying at least one video element of the environmentfeed and having a transparency ranging from zero percent transparent toone hundred percent transparent.
 10. The system of claim 9, theoperations further comprising: outputting at least one non-video elementcoordinated with the at least one video element.
 11. The system of claim9, wherein the environment feed comprises synthetic and naturalelements.
 12. The system of claim 9, the operations further comprising:tracking a user motion; and adapting the at least one video element onthe virtual window based on the user motion.
 13. The system of claim 9,the operations further comprising: altering at least one element of theenvironment feed over a period of time.
 14. The system of claim 9, theoperations further comprising: detecting a user in close proximity tothe virtual window; receiving an interaction from the user; and adaptingthe at least one video element on the virtual window based on theinteraction.
 15. The system of claim 10, the operations furthercomprising: retrieving a user profile; and selecting at least one of:the at least one video element and the at least one non-video element ofthe environment feed based on the user profile.
 16. The system of claim9, wherein the environment feed comprises at least one of: a soundelement, a video element, a smell element, a humidity element, atemperature element, and a wind element.
 17. A non-transitorycomputer-readable storage medium having stored therein instructionswhich, when executed by a computing device, cause the computing deviceto perform operations, the operations comprising: identifying anenvironment feed from a plurality of environment feeds, wherein eachenvironment feed of the plurality of environment feeds representsphysical environmental characteristics associated with a differentphysical location; and toggling a virtual window in a window casingbetween displaying at least one video element of the environment feedand having a transparency ranging from zero percent transparent to onehundred percent transparent.
 18. The non-transitory computer-readablestorage medium of claim 17, the operations further comprising:outputting at least one non-video element coordinated with the at leastone video element.
 19. The non-transitory computer-readable storagemedium of claim 17, wherein the environment feed comprises synthetic andnatural elements.
 20. The non-transitory computer-readable storagemedium of claim 17, the operations further comprising: tracking a usermotion; and adapting the at least one video element on the virtualwindow based on the user motion.